Video Upscaling Using Variational Methods

In this thesis we do image sequence upscaling using variational methods. We have developed, implemented and tested the three main elements of the upscaling part of a video processor using variational methods plus a non-variational preprocessing method detecting the scan format of the input video. The three upscalings needed are deinterlacing (DI), which is creating the never recorded every other line in interlaced video, video super resolution (VSR), which is increasing the spatial resolution of each frame in the video throughput, and temporal super resolution (TSR), which changes the frame rate of a sequence by creating fully new frames at the correct temporal positions. Our variational upscaling methods have been derived from a Bayesian inference framework for image sequence restoration and enhancement. The framework dictates simultaneous computation of the flow and intensities of the repaired and/or enhanced output sequences. The framework was first suggested for image sequence inpainting in [65]. From the framework we derive a motion adaptive (MA) deinterlacer and a motion compensated (MC) deinterlacer and test them together with a selection of known deinterlacers. To illustrate the need for MC deinterlacing the interlacing problem is introduced. It cannot be solved by MA deinterlacers or any simpler deinterlacers but only by MC deinterlacers. The major hurdle in doing MC deinterlacing is reliable optical flow computations on interlaced video. We discuss a number of strategies for computing optical flows on interlaced video hoping to shed some light on this problem. We produce results on real world video data with our variational MC deinterlacer that even in many difficult cases are indistinguishable from the ground truth. Producing high image quality on high definition (HD) displays when showing standard definition (SD) material is a problem of upscaling frame resolutions from low resolution (LR) to high resolution (HR) and is as such a super resolution (SR) problem, or as we prefer: Video super resolution (VSR). In technology available today the problem is typically solved using simple spatial interpolation. Using motion compensated methods instead will allow for information transport along the optical flow trajectories of the video and increase the level of detail and sharpness in the high resolution output. We present a variational motion compensated VSR method derived from our Bayesian framework that simultaneously computes the desired high resolution video and a high resolution flow field to increase accuracy of the temporal information transport. Creating super resolution flows has to our knowledge not been done before. Most advanced SR methods found in literature cannot be applied to general video with arbitrary scene content and/or arbitrary optical flows as it is possible with our simultaneous VSR method, which also allows for arbitrary discrete magnification factors. We show in test that our variational simultaneous VSR algorithm outperforms other SR methods applicable to our general video problem, and we also attempt to break the limits of super resolution [2] in our experiments by increasing the frame resolution eight times in both height and width (8x8 VSR). Temporal super resolution (TSR), the ability to convert video from one frame rate to another is a key functionality in a modern video processing systems. A different and often higher frame rate than what is recorded is desired for high frame rate displays, for video/film format conversion, or for super slow-motion.